Facial Expression Recognition Framework Research Articles

Contrastive Learning of View-invariant Representations for Facial Expressions Recognition

Although there has been much progress in the area of facial expression recognition (FER), most existing methods suffer when presented with images that have been captured from viewing angles that are non-frontal and substantially different from those used in the training process. In this article, we propose ViewFX, a novel view-invariant FER framework based on contrastive learning, capable of accurately classifying facial expressions regardless of the input viewing angles during inference. ViewFX learns view-invariant features of expression using a proposed self-supervised contrastive loss, which brings together different views of the same subject with a particular expression in the embedding space. We also introduce a supervised contrastive loss to push the learned view-invariant features of each expression away from other expressions. Since facial expressions are often distinguished with very subtle differences in the learned feature space, we incorporate the Barlow twins loss to reduce the redundancy and correlations of the representations in the learned representations. The proposed method is a substantial extension of our previously proposed CL-MEx, which only had a self-supervised loss. We test the proposed framework on two public multi-view facial expression recognition datasets, KDEF and DDCF. The experiments demonstrate that our approach outperforms previous works in the area and sets a new state-of-the-art for both datasets while showing considerably less sensitivity to challenging angles and the number of output labels used for training. We also perform detailed sensitivity and ablation experiments to evaluate the impact of different components of our model as well as its sensitivity to different parameters.

A self-attention-based fusion framework for facial expression recognition in wavelet domain

A self-attention-based fusion framework for facial expression recognition in wavelet domain

SAST: a suppressing ambiguity self-training framework for facial expression recognition

SAST: a suppressing ambiguity self-training framework for facial expression recognition

Multi-Relations Aware Network for In-the-Wild Facial Expression Recognition

Facial expression recognition (FER) becomes more challenging in the wild due to unconstrained conditions, such as the different illumination, pose changes, and occlusion of the face. Current FER methods deploy the attention mechanism in deep neural networks to improve the performance. However, these models only capture the limited attention features and relationships. Thus this paper proposes a novel FER framework called multi-relations aware network (MRAN), which can focus on global and local attention features and learn the multi-level relationships among local regions, between global-local features and among different samples, to obtain efficient emotional features. Specifically, our method first imposes the spatial attention on both the whole face and local regions to simultaneously learn the global and local salient features. After that, a region relation transformer is deployed to capture the internal structure among local facial regions, and a global-local relation transformer is designed to learn the fusion relations between global features and local features for different facial expressions. Subsequently, a sample relation transformer is deployed to focus on intrinsic similarity relationship among training samples, which promotes invariant feature learning for each expression. Finally, a joint optimization strategy is designed to efficiently optimize the model. The conducted experimental results on in-the-wild databases show that our method obtains the superior performance compared to some state-of-the-art models.

A high‐performance and lightweight framework for real‐time facial expression recognition

AbstractFacial expression recognition technology has become a powerful tool for conveying human emotions and intentions and is widely used in areas such as assisted driving and intelligent medical care. Due to the limited computing power of current hardware devices and the real‐time requirements of application scenarios, this paper proposes a high‐performance and lightweight framework for real‐time facial expression recognition framework to solve the problem of real‐time completion of expression recognition tasks under low hardware costs. To address these issues, this paper first designs a RepVGG and mobileNetV2 dual‐channel structure in the feature extraction. It is then input into the MobileViT Block for global feature modelling. Finally, the position vector of the capsule network is used to replace the output of the global pooling, preserving the spatial relationship of the salient features and enhancing the classification effect. Compared with the mainstream facial expression recognition algorithm that cannot get good classification results under low complexity conditions, the model has a significant accuracy improvement while ensuring lightweight. With only 294.60M FLOPS and 0.95M parameters, it achieved an accuracy of 97.53% on the KDEF dataset and 85.56% on the RAF‐DB, demonstrating the advanced nature of the algorithm.

Efficient facial expression recognition framework based on edge computing

Efficient facial expression recognition framework based on edge computing

LTVAL: Label Transfer Virtual Adversarial Learning framework for source-free facial expression recognition

LTVAL: Label Transfer Virtual Adversarial Learning framework for source-free facial expression recognition

A framework for facial expression recognition using deep self-attention network

A framework for facial expression recognition using deep self-attention network

Edge-AI-Driven Framework with Efficient Mobile Network Design for Facial Expression Recognition

Facial Expression Recognition (FER) in the wild poses significant challenges due to realistic occlusions, illumination, scale, and head pose variations of the facial images. In this article, we propose an Edge-AI-driven framework for FER. On the algorithms aspect, we propose two attention modules, Arbitrary-oriented Spatial Pooling (ASP) and Scalable Frequency Pooling (SFP), for effective feature extraction to improve classification accuracy. On the systems aspect, we propose an edge-cloud joint inference architecture for FER to achieve low-latency inference, consisting of a lightweight backbone network running on the edge device, and two optional attention modules partially offloaded to the cloud. Performance evaluation demonstrates that our approach achieves a good balance between classification accuracy and inference latency.

Geometry-Aware Facial Expression Recognition via Attentive Graph Convolutional Networks

Learning discriminative representations with good robustness from facial observations serves as a fundamental step towards intelligent facial expression recognition (FER). In this paper, we propose a novel geometry-aware FER framework to boost the FER performance based on both the geometric and appearance knowledge. Specifically, we propose an encoding strategy for facial landmarks, and adopt a graph convolutional network (GCN) to fully explore the structural information of the facial components behind different expressions. A convolutional neural network (CNN) is further applied to the whole facial observation to learn the global characteristics of different expressions. The features from these two networks are fused into a comprehensive high-semantic representation, which promotes the FER reasoning from both visual and structural perspectives. Moreover, to facilitate the networks to concentrate on the most informative facial regions and components, we introduce multi-level attention mechanisms into the proposed framework, which enhance the reliability of the learned representations for effective FER. Experiments on two challenging FER benchmarks demonstrate that the attentive graph-based learning on the facial geometry boosts the FER accuracy. Furthermore, the insensitivity of the geometric information to the appearance variations also improves the generalization of the proposed framework.

A Novel Efficient Patient Monitoring FER System Using Optimal DL-Features

Automated Facial Expression Recognition (FER) serves as the backbone of patient monitoring systems, security, and surveillance systems. Real-time FER is a challenging task, due to the uncontrolled nature of the environment and poor quality of input frames. In this paper, a novel FER framework has been proposed for patient monitoring. Preprocessing is performed using contrast-limited adaptive enhancement and the dataset is balanced using augmentation. Two lightweight efficient Convolution Neural Network (CNN) models MobileNetV2 and Neural search Architecture Network Mobile (NasNetMobile) are trained, and feature vectors are extracted. The Whale Optimization Algorithm (WOA) is utilized to remove irrelevant features from these vectors. Finally, the optimized features are serially fused to pass them to the classifier. A comprehensive set of experiments were carried out for the evaluation of real-time image datasets FER-2013, MMA, and CK+ to report performance based on various metrics. Accuracy results show that the proposed model has achieved 82.5% accuracy and performed better in comparison to the state-of-the-art classification techniques in terms of accuracy. We would like to highlight that the proposed technique has achieved better accuracy by using 2.8 times lesser number of features.

SMaTE: A Segment-Level Feature Mixing and Temporal Encoding Framework for Facial Expression Recognition.

Despite advanced machine learning methods, the implementation of emotion recognition systems based on real-world video content remains challenging. Videos may contain data such as images, audio, and text. However, the application of multimodal models using two or more types of data to real-world video media (CCTV, illegally filmed content, etc.) lacking sound or subtitles is difficult. Although facial expressions in image sequences can be utilized in emotion recognition, the diverse identities of individuals in real-world content limits computational models of relationships between facial expressions. This study proposed a transformation model which employed a video vision transformer to focus on facial expression sequences in videos. It effectively understood and extracted facial expression information from the identities of individuals, instead of fusing multimodal models. The design entailed capture of higher-quality facial expression information through mixed-token embedding facial expression sequences augmented via various methods into a single data representation, and comprised two modules: spatial and temporal encoders. Further, temporal position embedding, focusing on relationships between video frames, was proposed and subsequently applied to the temporal encoder module. The performance of the proposed algorithm was compared with that of conventional methods on two emotion recognition datasets of video content, with results demonstrating its superiority.

Hybrid Approach for Facial Expression Recognition Using Convolutional Neural Networks and SVM

Facial expression recognition is very useful for effective human–computer interaction, robot interfaces, and emotion-aware smart agent systems. This paper presents a new framework for facial expression recognition by using a hybrid model: a combination of convolutional neural networks (CNNs) and a support vector machine (SVM) classifier using dynamic facial expression data. In order to extract facial motion characteristics, dense facial motion flows and geometry landmark flows of facial expression sequences were used as inputs to the CNN and SVM classifier, respectively. CNN architectures for facial expression recognition from dense facial motion flows were proposed. The optimal weighting combination of the hybrid classifiers provides better facial expression recognition results than individual classifiers. The system has successfully classified seven facial expressions signalling anger, contempt, disgust, fear, happiness, sadness and surprise classes for the CK+ database, and facial expressions of anger, disgust, fear, happiness, sadness and surprise for the BU4D database. The recognition performance of the proposed system is 99.69% for the CK+ database and 94.69% for the BU4D database. The proposed method shows state-of-the-art results for the CK+ database and is proven to be effective for the BU4D database when compared with the previous schemes.

Automated Facial Expression Recognition Framework Using Deep Learning.

Facial expression is one of the most significant elements which can tell us about the mental state of any person. A human can convey approximately 55% of information nonverbally and the remaining almost 45% through verbal communication. Automatic facial expression recognition is presently one of the most difficult tasks in the computer science field. Applications of facial expression recognition (FER) are not just limited to understanding human behavior and monitoring person's mood and the mental state of humans. It is also penetrating into other fields such as criminology, holographic, smart healthcare systems, security systems, education, robotics, entertainment, and stress detection. Currently, facial expressions are playing an important role in medical sciences, particularly helping the patients with bipolar disease, whose mood changes very frequently. In this study, an algorithm, automated framework for facial detection using a convolutional neural network (FD-CNN) is proposed with four convolution layers and two hidden layers to improve accuracy. An extended Cohn-Kanade (CK+) dataset is used that includes facial images of different males and females with expressions such as anger, fear, disgust, contempt, neutral, happy, sad, and surprise. In this study, FD-CNN is performed in three major steps that include preprocessing, feature extraction, and classification. By using this proposed method, an accuracy of 94% is obtained in FER. In order to validate the proposed algorithm, K-fold cross-validation is performed. After validation, sensitivity and specificity are calculated which are 94.02% and 99.14%, respectively. Furthermore, the f1 score, recall, and precision are calculated to validate the quality of the model which is 84.07%, 78.22%, and 94.09%, respectively.

Facial expression recognition based on active region of interest using deep learning and parallelism.

The automatic facial expression tracking method has become an emergent topic during the last few decades. It is a challenging problem that impacts many fields such as virtual reality, security surveillance, driver safety, homeland security, human-computer interaction, medical applications. A remarkable cost-efficiency can be achieved by considering some areas of a face. These areas are termed Active Regions of Interest (AROIs). This work proposes a facial expression recognition framework that investigates five types of facial expressions, namely neutral, happiness, fear, surprise, and disgust. Firstly, a pose estimation method is incorporated and to go along with an approach to rotate the face to achieve a normalized pose. Secondly, the whole face-image is segmented into four classes and eight regions. Thirdly, only four AROIs are identified from the segmented regions. The four AROIs are the nose-tip, right eye, left eye, and lips respectively. Fourthly, an info-image-data-mask database is maintained for classification and it is used to store records of images. This database is the mixture of all the images that are gained after introducing a ten-fold cross-validation technique using the Convolutional Neural Network. Correlations of variances and standard deviations are computed based on identified images. To minimize the required processing time in both training and testing the data set, a parallelism technique is introduced, in which each region of the AROIs is classified individually and all of them run in parallel. Fifthly, a decision-tree-level synthesis-based framework is proposed to coordinate the results of parallel classification, which helps to improve the recognition accuracy. Finally, experimentation on both independent and synthesis databases is voted for calculating the performance of the proposed technique. By incorporating the proposed synthesis method, we gain 94.499%, 95.439%, and 98.26% accuracy with the CK+ image sets and 92.463%, 93.318%, and 94.423% with the JAFFE image sets. The overall accuracy is 95.27% in recognition. We gain 2.8% higher accuracy by introducing a decision-level synthesis method. Moreover, with the incorporation of parallelism, processing time speeds up three times faster. This accuracy proves the robustness of the proposed scheme.

Double OptconNet architecture based facial expression recognition in video processing

ABSTRACT A Double optimization based convolution network model is introduced in the proposed video based facial expression recognition framework. The proposed model comprises U-shaped network, Residual-Network architecture, and Coot optimization. Before performing expression recognition, the input video is subjected to pre-processing, and face detection is performed over the extracted frames using the viola jones algorithm. The U-shaped network has the advantage of improving the processing speed of the convolution network, whereas the residual network can reduce the error that occurs during the frame encoding and gradient dissipation avoidance. Due to this merit, these two networks are combined and introduced in the proposed framework for facial expression recognition. The experimental evaluation is performed using a matrix laboratory tool over the three datasets: Affectiva-MIT Facial Expression Dataset, BAUM-1s and Real-world affective faces database. The comparative analysis shows that the proposed network has attained an efficient recognition rate than other existing network architecture.

Efficient Recognition of Facial Expression with Lightweight Octave Convolutional Neural Network

In various social activities, people usually focus on the other person’s facial expression, which is an important element in interpersonal communication. The facial expression typically reflects a person’s current mood and conveys emotional information. Perceiving the facial expressions of people in images through cameras has always been a popular research topic. Previous studies have classified facial expressions into different categories, such as happy, sad, fear, angry, calm, disgust and surprise, and have identified them using image processing methods. However, traditional image processing methods have a low detection efficiency and low recognition accuracy due to variation of perspectives. As a result, most of them can only be applied to the front face and short distance situations. In this paper, we propose a lightweight deep learning framework for facial expression recognition using Octave convolutional neural network (FerOctNet). FerOctNet including multi-scale convolutional processing and residual learning is able to obtain multi-scale features with enriched representation ability by integrating the deep level features with rich semantic information with shallow details of the features. Compared with other deep learning networks, not only does the proposed method have a good recognition rate, but also contains fewer parameters in the network.

Multi-Pose Facial Expression Recognition Using Hybrid Deep Learning Model with Improved Variant of Gravitational Search Algorithm

The recognition of human facial expressions with the variation of poses is one of the challenging tasks in real-time applications such as human physiological interaction detection, intention analysis, marketing interest evaluation, mental disease diagnosis, etc. This research work addresses the problem of expression recognition from different facial poses at the yaw angle. The major contribution of the paper is the proposal of an autonomous pose variant facial expression recognition framework using the amalgamation of a hybrid deep learning model with an improved quantum inspired gravitational search algorithm. The hybrid deep learning model is the integration of the convolutional neural network and recurrent neural network. The applicability of the hybrid deep learning model can be considered as significant if the feature set is efficiently optimized to have the discriminative features respective to each expression class. Here, the Improved Quantum Inspired Gravitational Search Algorithm (IQI-GSA) is utilized for the selection and optimization of features. The IQI-GSA method is significant for optimizing the features compared to quantum-behaved binary gravitation search algorithm for handing the local optima and stochastic characteristics. Comparing with state-of-art techniques, the proposed framework exhibits the outperformed recognition rate for experimentation on Karolinska Directed Emotional Faces (KDEF) and Japanese Female Facial Expression (JAFFE) datasets.

Facial expression recognition with grid-wise attention and visual transformer

Facial Expression Recognition (FER) has achieved remarkable progress as a result of using Convolutional Neural Networks (CNN). Relying on the spatial locality, convolutional filters in CNN, however, fail to learn long-range inductive biases between different facial regions in most neural layers. As such, the performance of a CNN-based model for FER is still limited. To address this problem, this paper introduces a novel FER framework with two attention mechanisms for CNN-based models, and these two attention mechanisms are used for the low-level feature learning the high-level semantic representation, respectively. In particular, in the low-level feature learning, a grid-wise attention mechanism is proposed to capture the dependencies of different regions from a facial expression image such that the parameter update of convolutional filters in low-level feature learning is regularized. In the high-level semantic representation, a visual transformer attention mechanism uses a sequence of visual semantic tokens (generated from pyramid features of high convolutional layer blocks) to learn the global representation. Extensive experiments have been conducted on three public facial expression datasets, CK+, FER+, and RAF-DB. The results show that our FER-VT has achieved state-of-the-art performance on these datasets, especially with a 100% accuracy on CK + datasets without any extra training data.

Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network.

Facial expression recognition has been an active area of research over the past few decades, and it is still challenging due to the high intra-class variation. Traditional approaches for this problem rely on hand-crafted features such as SIFT, HOG, and LBP, followed by a classifier trained on a database of images or videos. Most of these works perform reasonably well on datasets of images captured in a controlled condition but fail to perform as well on more challenging datasets with more image variation and partial faces. In recent years, several works proposed an end-to-end framework for facial expression recognition using deep learning models. Despite the better performance of these works, there are still much room for improvement. In this work, we propose a deep learning approach based on attentional convolutional network that is able to focus on important parts of the face and achieves significant improvement over previous models on multiple datasets, including FER-2013, CK+, FERG, and JAFFE. We also use a visualization technique that is able to find important facial regions to detect different emotions based on the classifier’s output. Through experimental results, we show that different emotions are sensitive to different parts of the face.